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INTRODUCTION 



The AEL Measure of Academic Supportiveness and Climate (AEL MASC) was 
developed as part of the MAACK Pilot Schools project currently underway at AEL. MAACK 
stands for Maximizing Achievement for African American Children in Kanawha. The AEL 
MASC was designed to determine students’ perceptions about themselves as students and about 
their school experiences. The survey also examines students’ perceptions of their families’ 
awareness of and involvement in their children’s school lives. All of these factors are strongly 
related to student performance; if students do not feel they have a supportive school environment 
as well as family support, they are less likely to perform well in school. Finally, the survey is 
designed to facilitate comparisons among the perceptions of groups of students according to their 
gender or ethnicity. 

AEL MASC survey items were developed after a review of the literature revealed a lack 
of validated student surveys addressing issues of school and family support and fairness in tenns 
of instructional opportunities and extracurricular activities for all students. In particular, the 
following surveys were identified and deemed to be inappropriate for this particular project: 

★ Gottfredson Associates (2001). This School Diversity Inventory measures 
perceptions of administrative leadership in promoting a climate of respect for 
diversity. Although several subscales were relevant, this inventory was rejected due 
to associated costs for utilizing materials and scoring. 

★ Wayman (2002). This survey assessed student perceptions of teacher bias in terms 
of Mexican American and non-Latino White students. Reliability and validity 
information was not provided, and the instrument did not focus on perceptions of bias 
toward African American students. 

★ Pohan and Aguilar (2001). This survey assessed educators’ beliefs about diversity 
in personal and professional contexts. While this instrument possessed sufficient 
reliability and validity, it focused exclusively on teachers’ perceptions. 

★ Rothfarb (1992). This survey provided baseline data on secondary students’ 
perceptions of intercultural relations. While possessing sufficient reliability and 
validity, the survey was less appropriate for elementary students. 

★ Walberg, Anderson, and Cayne (1969). This survey includes subscales measuring 
students’ perceptions of diversity, environment, favoritism, and democratic climate. 
However, reviews suggest less than sufficient reliability. 
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The AEL MASC began in 2002 as a 27-item student survey (the MAACK Student 
Survey) developed by AEL staff providing technical assistance to schools involved in the 
MAACK project. After minor revisions in 2003 to improve item clarity, the instrument was pilot 
tested with 2,818 students (49% boys, 51% girls) in eight schools from grades 3 through 1 1 (four 
MAACK pilot schools and four comparison schools). Approximately 21% of the sample 
students were African American, 66% were White, and the remaining 13% were members of 
other racial/ethnic groups. Sixty-eight percent of the respondents were enrolled in grades 9 
through 12; the remaining respondents were enrolled fairly evenly in grades 3 through 8 
(approximately 7% per grade). More than half of the participants (57%) attended schools 
actively participating in the MAACK pilot schools project. 

Results of a preliminary factor analysis of the pilot-test data revealed five subscales, 
which together explained approximately 5 1% of the variance in respondents’ scores. These 
subscales include: 

★ Staff Supportiveness, This eight-item subscale assesses the degree to which school staff 
know and respect students, and treat them fairly. It explains 19.3% of the variance and 
has a Cronbach Alpha reliability coefficient of .89. 

★ Family Expectations, This five-item subscale measures the extent to which students feel 
their families are aware of their progress in school and expect and support their academic 
success. It explains 1 1 . 1% of the variance and has a Cronbach Alpha reliability 
coefficient of .66. 

★ Welcoming Environment. This six-item subscale assesses students’ perceptions of the 
degree to which their school climates are both intellectually and socially safe and 
comfortable. It explains 10.1% of the variance and has a Cronbach Alpha reliability 
coefficient of .79. 

★ School Ties, This three-item subscale measures family presence in the school and 
student participation in extracurricular activities. It explains 5.4% of the variance and has 
a Cronbach Alpha reliability coefficient of .50. 

★ Student Academic Efficacy. This two-item subscale assesses students’ sense of their 
academic capability. It explains 5.2% of the variance and has a Cronbach Alpha 
reliability coefficient of .37. 

In an effort to increase the less than satisfactory reliability coefficients of some subscales 
and to balance more evenly the number of items comprising each subscale, the instrument was 
again revised in 2003. Fifteen new items were added, and the instrument was formally renamed 
the AEL Measure of Academic Supportiveness and Climate (AEL MASC). This version of the 
AEL MASC was used in the data collection for the field test. 

This paper provides a summary of the field test of the AEL Measure of Academic 
Supportiveness and Climate (AEL MASC) student instrument. Findings from the field test can 
be utilized in revising and fine-tuning the instrument for future administrations. 
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METHODS 



Participants 



Eight schools participating in AEL’s Maximizing Achievement for African American 
Children in Kanawha (MAACK) County Pilot Schools project (four pilot schools and four 
comparison schools) were required by the district superintendent to administer the AEL MASC 
to all students in grades 3 through 12 in late April and early May of 2003. Included in this group 
were four elementary schools, two middle schools, and two high schools. Half of the schools at 
each level (pilot schools) were working with AEL staff to implement new instructional policies 
and practices designed to help narrow the achievement gaps for minority and low-income 
students. Table 1 provides the numbers of students enrolled in grades 3 through 12 and surveys 
delivered to each school. Extra surveys were provided to each school to compensate for possible 
inaccuracies in the most recent enrollment data available. 

Table 1 : Enrollment and Number of Surveys Distributed to Participating Schools 



School 


Number of 
Students 
Enrolled 


Number of 
Surveys 
Distributed 


Comparison Elementary School A 


179* 


150 


Comparison Elementary School B 


366 


420 


Comparison Middle School 


471 


525 


Comparison High School 


1,100 


1,260 


Pilot Elementary School A 


215* 


210 


Pilot Elementary School B 


264* 


240 


Pilot Middle School 


692 


805 


Pilot High School 


1,454 


1,680 


Total 


4,741 


5,290 



* Surveys were not administered to students below grade three, hence the number 
of students enrolled is larger than the number of surveys distributed. 



As shown in Table 2, the overall number of surveys administered in 2003 was much 
lower than the number of students enrolled in grades 3 through 12 due to survey administration 
difficulties at each school (described elsewhere); further, a significant proportion of the returned 
surveys were incomplete. However, the overall return rate in 2003 (56.42%) was very similar to 
that obtained in the same schools in 2002 (54.61%), and all but one school returned more surveys 
in 2003 than in 2002. 

For test-retest purposes, the survey was administered a second time in three of the eight 
schools. Although all of the comparison schools in the MAACK pilot schools project were 
contacted and offered a monetary bonus for participating in the test-retest portion of the field 
test, only one elementary and the middle school agreed to take part. Subsequently, the pilot high 
school was contacted with the same offer and agreed to take part. 
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Table 2: Number of Complete, Incomplete, and Total Surveys Administered 



School 


Number of Surveys 
Administered in 2003 


Total Number of Surveys 
Administered 


Complete 


Incomplete 


2003 


2002 


Comparison Elementary School A 


83 


0 


83 


77 


Comparison Elementary School B 


297 


9 


306 


284 


Comparison Middle School 


317 


17 


334 


237 


Comparison High School 


724 


41 


765 


577 


Pilot Elementary School A 


74 


2 


76 


69 


Pilot Elementary School B 


124 


4 


128 


109 


Pilot Middle School 


397 


24 


421 


369 


Pilot High School 


448 


114 


562* 


867 


Total 


2,464 


211 


2,675 


2,589 



* Staff at Pilot High School made the decision not to survey students in the 12th grade. 



Students were not asked to provide an identification number on the AEL MASC. Instead, 
AEL staff assigned an identification number to each survey after its completion and return. 

These identification numbers consisted of 14 numbers and were developed by using “39” to 
identify the county (the state-level code for Kanawha County); a four-digit number to identify 
the school based on the first letter of each word in the school name (i.e., Pilot High School = 

0747 — first a zero to indicate the school name contains only three words, followed by numbers 
matching the letters on a telephone number pad); a four-digit number for the month and year in 
which the survey was administered (i.e., April 2003 = 0403); and a four-digit number providing a 
consecutive count of returned surveys (i.e., 0001, 0002, etc.). Each of the completed and 
returned test-retest surveys packaged together was given the same identification number, 
regardless of whether or not the demographic infonnation matched. (The “Procedures” section 
explains how each student’s test and retest were matched without violating confidentiality.) 

As shown in Table 3, the overall return rate (30.03%) in this portion of the field test was 
much lower than the overall survey return rates in 2002 and 2003. In addition, approximately 
15% of the survey pairs returned were incomplete. 

Table 3: Number of Complete and Incomplete Pairs of Surveys 
for Each School Participating in the Test-Retest Administration 



School 


Complete 


Incomplete 


Total 


Comparison Elementary School B 


40* 


6 


46 


Comparison Middle School 


278 


16 


294 


Pilot High School 


348 


98 


446 


Total 


666 


120 


786 



*Due to error in administering the surveys at this school, only 46 paired surveys were returned. 
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During the inspection of the test-retest surveys for completion, staff noticed that the 
demographic data on some of the physically-paired surveys did not match. For instance, for one 
pair, demographics on the first administration indicate that the respondent was a 9th-grade White 
girl enrolled at the school for six months to one school year, but on the second survey, the 
demographics indicate that the respondent was an 1 lth-grade Hispanic boy enrolled at the school 
for more than one year. Staff subsequently examined the handwriting on the surveys. Physical 
pairs with discrepant demographic data were sorted into those with similar handwriting and those 
with either dissimilar handwriting or a blank demographic section. Surveys with similar 
handwriting were treated as genuine pairs, but only the demographics on the first administration 
were used to describe the respondent. A variable indicating these changes was added to the 
database for use in subsequent analyses. In cases in which survey handwriting could not be 
matched, the data were not included in the test-retest analyses, although data from the first 
survey in an unmatched pair were used to establish validity. Table 4 provides a breakdown of 
the number of surveys for which the demographic information either did not match or was blank. 

Table 4: Number of Survey Pairs with Discrepant Demographic 
Information, Matched and Unmatched Handwriting 



School 


Similar 

Handwriting 


Different 
Handwriting/ 
Can’t Tell 


Blank 

Demographic 

Section 


Total 


Comparison Elementary School A 


9 


10 


4 


23 


Comparison Middle School 


65 


23 


8 


96 


Pilot High School 


46 


27 


65 


138 


Total 


120 


60 


77 


257 



School Sample Descriptions 

Comparison Elementary School A. Of the 83 respondents, 48% were female and 52% 
male. More than half (52%) indicated they were White, and 30% selected African American. 
Nearly half of the respondents (45%) were in the third grade, 23% were in the fourth grade, and 
32% were in the fifth grade. 

Comparison Elementary School B. Of the 306 respondents, 50% were male and 50% 
female. More than half (54%) were White, and 32% were African American. Respondents were 
grouped fairly evenly among the third grade (29%), fourth grade (38%), and fifth grade (33%). 

Comparison Middle School. Of the 334 respondents, 52% were female and 48% male. 
More than half (61%) were White, and 26% were African American. More than a third (36%) 
were in the sixth grade, 40% were in the seventh grade, and 24% were in the eighth grade. 

Comparison High School. Of the 765 respondents, 54% were female and 46% male. 
More than three fourths (76%) were White, and 15% were African American. More than a third 
(34%) were in the 9th grade, 31% were in the 10th grade, 20% were in the 1 1th grade, and 15% 
were in the 12th grade. 
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Pilot Elementary School A. Of the 76 respondents, 50% were female and 50% male. 
Nearly three fourths (71%) were African American, and 17% were White. Respondents were 
grouped fairly evenly among third grade (35%), fourth grade (39%), and fifth grade (26%). 

Pilot Elementary School B. Of the 128 respondents, 44% were female and 56% male. 
Nearly half (49%) were African American, and 31% were White. Almost half (40%) were in 
third grade, 28% were in fourth grade, and 31% were in fifth grade. 

Pilot Middle School. Of the 421 respondents, 57% were female and 43% male. More 
than half (59%) were White, and 24% were African American. Respondents were divided fairly 
evenly among sixth grade (36%), seventh grade (36%), and eighth grade (28%). 

Pilot High School. Of the 562 respondents, 49% were female and 51% male. Nearly 
three fourths (71%) were White, and 18% were African American. Respondents were grouped 
fairly evenly among the 9th grade (32%), 10th grade (33%), and 1 1th grade (34%). 



Measures 



AEL MASC. The version of the AEL MASC used for the field test included 42 survey 
items plus 5 demographic items (student gender, ethnicity, grade level, length of time at school, 
and school name). It was hypothesized that the 42 survey items would be subsumed within the 
five subscales identified in the pilot test. The expected subscales and relevant items included the 
following: Staff Supportiveness - 8 items, Family Expectations - 10 items, Welcoming 
Environment - 8 items, School Ties - 8 items, and Student Academic Efficacy - 8 items. 

The response options for the 42 items were based on a Likert-type 1 to 5 truth scale: 

1 = Not at all, 2 = Rarely true, 3 = Sometimes true, 4 = Often true, 5 = Always true. A sixth 
option of “Don’t know” was also provided. 

Miami-Dade School Climate Survey. As a measure of concurrent validity, the Miami- 
Dade School Climate Survey was identified and secured to include on the field test of the AEL 
MASC. The survey was developed and is administered annually by Miami-Dade County 
(Florida) Public Schools. The student form of this instrument consists of 27 items structured as 
statements about the school. Respondents indicate the extent to which they agree or disagree 
with each of 26 items using a 5 -point scale ranging from 1 = Strongly Disagree to 5 = Strongly 
Agree and a midpoint of 3 = Undecided/Unknown. The 27th item asks students to give their 
school an overall grade of A, B, C, D, or F, and is used as an overall summary of an individual’s 
item responses (Gomez & Shay, 2001). Staff and parent forms of the survey have also been 
developed; parent forms are available in English, Spanish, and Haitian Creole. The instrument 
has been shown to have good internal consistency (Cronbach Alpha > .90) and to have face 
validity with stakeholders including parents, school and district staff, and students. For use in 
the concurrent validity study, the 27th student item (overall school rating) was excluded. 
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Procedures 



The two surveys (the AEL MASC and the Miami-Dade School Climate Survey) were 
merged into one four-page scannable survey for ease of administration. AEL staff sent an 
explanatory letter to the district superintendent and also provided a draft cover letter for his 
signature infonning the eight principals that their schools were required to administer the AEL 
MASC as part of Kanawha County’s ongoing collaboration with AEL in the MAACK Pilot 
Schools effort. The letter instructed elementary school principals to have teachers administer the 
MASC in grades three and higher. Middle and high school principals were instructed to have the 
survey administered in English/Language Arts classes to ensure that all students attending the 
school would have an opportunity and sufficient time to complete the MASC. (However, the 
surveys at Pilot High School were administered during homeroom instead.) Principals were 
asked to ensure that surveys were administered during the week of May 9, 2003, and to contact 
AEL following the administration to arrange for the completed surveys and any remaining 
unused materials to be collected. 

Surveys were bundled into classroom-sized sets of 30 along with one instruction card for 
teachers in each bundle. The instructions for teachers administering the surveys were printed on 
heavy red card stock and surveys were printed on pink paper. Each bundle of surveys and 
instructions was placed in a 10 x 13 envelope. These envelopes were then placed in boxes 
labeled for each of the schools, along with extra copies of the survey bound with a rubber band. 

Procedures for the three schools participating in the retest portion of the field test differed 
slightly because teachers were asked to administer the survey on two occasions approximately 
two weeks apart and because surveys had to be paired for each student without sacrificing 
confidentiality. Accordingly, teachers in these schools were asked to have students complete the 
pink AEL MASC survey, place the completed survey in a 9 x 12 envelope, seal the envelope, 
and write their name across the seal on the back. Teachers were asked to collect these envelopes 
and keep them in a safe place for approximately two weeks. At that time, teachers were to return 
each sealed envelope to the appropriate student and instruct students to open the envelope, 
remove the previously completed pink survey, and seal it in a new 9x12 envelope so that 
students’ names were no longer associated with the data. Teachers were also asked to request 
that students avoid looking at their original survey responses during this process. 

Next, teachers were asked to distribute the green copies of the AEL MASC and have 
students complete it. Afterward, each student was to seal this green copy and the 9 x 12 envelope 
containing the previously completed pink survey in a large (10 x 13) envelope. Teachers were 
asked to collect these large envelopes, seal them in the expandable white envelope provided, and 
return the packet of completed surveys to the school principal. Finally, teachers were asked to 
ensure that envelopes with names on them were empty and then destroy them. 

For these three schools, sets of 30 pink and 30 green surveys were bundled along with 60 
small 9 x 12 envelopes, 30 large 10 x 13 envelopes, one large expandable white envelope, and 
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one letter and instruction card for teachers. Again, these bundles were boxed and labeled for 
each of the schools, along with extra envelopes in both sizes and extra copies of the survey. 



Staff from the four pilot schools received the survey materials for their schools at a 
MAACK Pilot Schools team meeting held at AEL’s office on May 5 (given the larger number of 
students, AEL staff delivered the surveys to Pilot High School). On the next business day, an 
AEL staff member delivered the survey materials to each of the comparison schools. 

The SEDCAR Standards Checklist was used to document the data collection methods 
used in this field test (Cooperative Education Data Collection and Reporting [CEDCAR] 
Standards Project Task Force, 1991). 



Analyses 



Surveys were scanned into school files using Remark software. School files were then 
exported to SPSS statistical software for cleaning, merging, and subsequent analyses. 
Descriptive statistics for the AEL MASC were generated by item for initial data exploration. 

All “don’t know” responses were excluded from subsequent analyses. Only three items had this 
option selected by more than 10% of respondents (13%, 16%, 20%); two thirds of the items had 
the “don’t know” option chosen by 5% or less. All analyses excluded missing responses. 

Concurrent validity was established via a Pearson product moment correlation between 
the overall scores of the AEL MASC and the Miami-Dade School Climate Survey. Construct 
validity was measured via factor analysis of the 42 AEL MASC items. Internal consistency of 
the AEL MASC was measured via Cronbach Alpha reliability coefficients by subscales and for 
the total instrument; the internal consistency of the overall Miami-Dade School Climate Survey 
also was generated. Test-retest reliability for the AEL MASC was measured via Pearson 
correlations for each factor between test and retest scores for the full group and by subgroups. 
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FINDINGS 



Validity 



Construct* 

According to Gable and Wolf (1993), factor analysis is used to examine “empirically the 
interrelationships among the items and to identify or verify clusters of items that share sufficient 
variation to justify their existence as a factor or construct to be measured by the instrument” (p. 
108). The merged data file, with 1,008 cases (factor analysis excluded those 1,667 cases with 
missing responses to any of the 42 items on the AEL MASC), has a variable to case ratio of 24: 1, 
well above the authors’ minimal recommendation of 10:1. 

A principal-component nonorthogonal oblique factor analysis was generated that retained 
factor eigenvalues greater than or equal to 1.0. The oblique method was chosen on the 
assumption that subsequent factors would be correlated with one another. As recommended by 
Gable and Wolf (1993), an orthogonal varimax factor analysis (independent factors) was also 
generated for comparison purposes. The oblique analysis resulted in four factors plus the 
beginnings of two additional factors; the varimax analysis resulted in five factors. For 
comparison purposes, both methods were again employed in generating a four- factor forced 
model. Inspection of all of these analyses revealed that the original oblique technique with 
eigenvalues greater than or equal to 1.0 resulted in the best-fitting model. The following results 
are based on this analysis. 

The oblique factor analysis generated six factors; rotation converged in 1 1 iterations. The 
six rotated factor eigenvalues (in factor order) were 1 1.66, 7.22, 7.28, 7.23, 1.59, and 2.74. 
Inspection of the pattern matrix revealed that some items loaded on multiple factors, but most of 
these multiple loadings were below .300, indicating a much stronger loading on a single factor. 

Table 5 presents the pattern matrix from this factor analysis. 

The fifth and sixth factors seemed to be mere beginnings of factors, even with 
eigenvalues above 1.0. The fifth factor was composed of two items, with loading values of .561 
and .441. However, one of these items also loaded on the fourth factor at .338. The sixth factor 
was composed of only one item (loading value of .698), which did not load on any other factor at 
.300 or above. Therefore, these two factors were excluded from subsequent analyses. 

The oblique factor analysis does not generate the percent of variance accounted for after 
rotation. However, Gable and Wolf (1993) suggest “the variance accounted for after rotation 
will be the same for each factor for the oblique and varimax solutions” (p. 127). Therefore, both 
of the varimax analyses were inspected. The unrestricted-factor model accounted for 52% of the 
variance; the four-factor model, 49%. Since the order of the third and fourth factors fluctuated 
between oblique and varimax rotations, individual factor percentages are not reported. 



*Although reliability is a prerequisite for validity, the results from the construct validity (factor analysis) 
are presented first so that the identified factors can be used in subsequent internal consistency and test- 
retest reliability analyses. 
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Table 5: Pattern Matrix for Oblique Factor Analysis of AEL MASC 





Faci 


tors 


Items 


1 


2 


3 


4 


5 


6 


8 


.825 












13 


.824 












40 


.806 










.128 


22 


.797 


.137 










42 


.758 










-.124 


17 


.706 






-.119 




.111 


35 


.643 






.216 






37 


.639 




.143 


.124 


-.248 


-.110 


12 


.528 




.144 


.277 


-.185 




24 


.503 


.179 




.209 






27 


.499 










.291 


33 


.488 




.171 


.246 


-.317 




7 


.479 




.223 


.177 


-.173 


-.117 


14 


.468 






.203 


.178 


.133 


32 


.437 




.220 






.110 


18 


.420 




.107 


.303 


-.145 


-.153 


28 


.355 


.142 


.149 


.227 


-.204 


.115 


3 


.353 


.188 


.127 


.228 


-.276 




2 


.345 




.108 






.334 


26 




.842 










38 




.824 










11 




.752 










41 




.731 






.104 




36 




.720 




.197 






31 




.531 


.198 




.130 




19 




-.185 


.801 






-.215 


15 




.161 


.690 




.130 




39 


.119 


.185 


.591 








4 






.570 




.219 


.295 


34 


.132 


.156 


.522 


.123 


-.127 




30 




.415 


.483 








10 


.239 


.146 


.348 








5 








.753 


.215 




16 








.658 


i 

oo 


.216 


9 








.625 


.437 


-.138 


6 




.229 




.465 


-.133 


.172 


21 




.351 




.398 




.215 


23 


.394 




.105 


.397 


-.117 


.136 


25 


.149 


.178 


.174 


.270 


-.221 


.200 


20 






.165 




.561 


.221 


29 


.150 


.252 




.338 


.441 


-.208 


1 








.133 


.111 


.698 
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To further substantiate the four-factor model, the component correlation matrix was 
inspected. The matrix in Table 6 shows that the first four components (factors) all correlate with 
each other above .30; the fifth and sixth factors have no correlations at this level. 



Table 6: Component Correlation Matrix for Oblique Factor Analysis of AEL MASC 



Component 


Factor 1 


Factor 2 


Factor 3 


Factor 4 


Factor 5 


Factor 6 


1 


1.00 












2 


.32 


1.00 










3 


.45 


.32 


1.00 








4 


.45 


.31 


.33 


1.00 






5 


-.08 


.04 


.01 


-.06 


1.00 




6 


.20 


.28 


.19 


.10 


.02 


1.00 



Finally, the four factors were named. Gable and Wolf (1993) suggest that items loading 
at .400 or above contribute most to naming the factors and should be studied most closely in 
interpreting the factor. After reviewing the analysis, the four factors were named: Factor 1 - 
Student Belonging, Factor 2 - Family Expectations, Factor 3 - Student Academic Efficacy, and 
Factor 4 - Family/School/Student Involvement. 

After the factors were named, subscales were created in SPSS. Because the subscales 
had different numbers of items, item-level subscale means (total subscale score divided by 
number of items in the subscale) were used to enable cross-subscale comparisons. Table 7 
depicts the descriptive statistics (number, mean, and standard deviation) for the four subscales 
for the full group and by subgroups. Figures 1 through 4 present bar charts of the four subscales 
by the full group and by subgroups (gender, ethnicity, and building level). 



Table 7: Descriptive Statistics by Subscale for Full Group 
and Subgroups for AEL MASC 



Sub- 


Sta- 


Full 


Gender 


Ethnicity 


Building Level 


scales 


tistics 


Group 


Male 


Female 


White 


Af. Am. 


Elem. 


Mid. 


High 


Student 


n 


2,656 


1,214 


1,316 


1,610 


586 


590 


755 


1,311 


Belonging 


mean 


3.54 


3.50 


3.60 


3.52 


3.67 


4.06 


3.51 


3.33 




SD 


0.80 


0.81 


0.77 


0.78 


0.76 


0.69 


0.77 


0.75 


Family 


n 


2,636 


1,207 


1,313 


1,604 


585 


590 


752 


1,294 


Expec- 


mean 


4.50 


4.41 


4.60 


4.51 


4.56 


4.63 


4.58 


4.39 


tations 


SD 


0.69 


0.74 


0.59 


0.66 


0.62 


0.60 


0.58 


0.77 


Student 


n 


2,652 


1,212 


1,316 


1,609 


586 


590 


754 


1,308 


Academic 


mean 


3.74 


3.72 


3.79 


3.74 


3.84 


3.97 


3.77 


3.62 


Efficacy 


SD 


0.74 


0.75 


0.69 


0.72 


0.68 


0.71 


0.72 


0.73 


Family/ 


n 


2,651 


1,213 


1,315 


1,608 


586 


589 


755 


1,307 


School/ 


mean 


3.81 


3.77 


3.88 


3.79 


3.96 


4.36 


3.95 


3.49 


Student 

Involve. 


SD 


0.82 


0.83 


0.80 


0.81 


0.77 


0.62 


0.76 


0.79 
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Figure 1: AEL MASC Subscale Means for Full Group 
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Figure 2: AEL MASC Subscale Means by Gender 
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Figure 3: AEL MASC Subscale Means by Ethnicity 
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Figure 4: AEL MASC Subscale Means by Building Level 
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Concurrent 



As a measure of concurrent validity, the AEL MASC scores were correlated with those of 
the Miami-Dade School Climate Survey, an instrument with previously established reliability 
and validity in assessing school climate from the students’ perspective. Table 8 shows the results 
of Pearson product moment correlations for the totaled 42 items on the AEL MASC with the 
totaled 26 items on the Miami-Dade School Climate Survey for the full group and by subgroups. 
Except for the high school analysis, all correlations were in the .50 range, indicating a moderate 
relationship (Hinkle, Wiersma, & Jurs, 1998). All analyses were significant at the .01 level, 
which is not unexpected given the large sample size. 

Table 8: Concurrent Pearson Product Moment Correlations by Full Group and Subgroups 
for Total Score on AEL MASC and Miami-Dade School Climate Survey 





Full 


Gender 


Ethnicity 


Building Level 




Group 


Males 


Females 


White 


Af. Am. 


Elem. Mid. 


High 


Total score 
on both 


.56* 


.56* 


.57* 


.58* 


.54* 


.52* ! .58* 


.49* 


surveys 


n=2,599 


n= 1,204 


n=l,312 


n= 1,604 


n=583 


n=589 n=743 


n=l,267 



* Significant at .01 



Reliability 



Internal Consistency 

As a measure of the internal consistency of the overall Miami-Dade School Climate 
Survey, Cronbach Alpha reliability coefficients for this administration were generated for the full 
group and by gender, ethnicity (White and African American), and building level (elementary, 
middle, and high school). Table 9 shows the overall reliability was high for this administration 
of the instrument, ranging from .84 to .91 for this set of scores. 

Table 9: Cronbach Alpha Reliability Coefficients by Full Group and Subgroups 
for Miami-Dade School Climate Survey 





Full 


Gender 


Ethnicity 


Building Level 




Group 


Males 


Females 


White 


Af. Am. 


Elem. 


Mid. 


High 


All items on 
Miami-Dade 


.90 


.91 


.89 


.90 


.88 


.84 


.89 


.90 


School 

Climate 

Survey 


n=2,165 


n= 1,024 


n= 1,113 


n= 1,393 


n=462 


n=426 


n=628 


n=l,l 1 1 
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As a measure of the internal consistency of the overall AEL MASC and its four identified 
factors, Cronbach Alpha reliability coefficients were generated for the full group and by gender, 
ethnicity (White and African American), and building level (elementary, middle, and high 
school) for this administration. Table 10 shows that the overall internal consistency reliability 
was very high, ranging only marginally from .93 to .95 across the various breakdowns. Of the 
four factors, Student Belonging was most reliable, with coefficients ranging across breakdowns 
from .90 to .94. The remaining three factors were slightly less reliable, with coefficients ranging 
from .64 to .84. Of the 24 cells, 18 had coefficients in the .70’s, 5 were in the ,80’s, and 1 was in 
the AO’s (middle school breakdown for Family Expectations). 

Table 10: Cronbach Alpha Reliability Coefficients by Full Group 
and Subgroups for AEL MASC 





Full 

Group 


Gender 


Ethnicity 


Building Level 


Males 


Females 


White 


Af. Am. 


Elem. 


Mid. 


High 


All items on 
AEL MASC 


.95 

n=l,008 


.95 

n=434 


.95 

n=550 


.95 

n=678 


.94 

n=188 


.93 

n=198 


.94 

n=239 


.94 

n=571 


Factor 1: 

Student 

Belonging 


.93 

n= 1,337 


.93 

n=593 


.94 

n=707 


.94 

n=866 


.92 

n=268 


.90 

n=308 


.93 

n=336 


.93 

n=693 


Factor 2: 

Family 

Expectations 


.79 

11=2, 111 


.80 

n=954 


.77 

n= 1,091 


.80 

n=l,337 


.75 

n=456 


.75 

n=447 


.64 

n=592 


.84 

n=l,072 


Factor 3: 
Student 
Academic 
Efficacy 


.78 

n=2,007 


.79 

n=912 


.78 

n= 1,042 


.80 

n= 1,287 


.70 

n=427 


.73 

n=406 


.78 

n=573 


.79 

n=l,028 


Factor 4: 
Family/SchooP 
Student 
Involvement 


.79 

n=l,988 


.79 

n=898 


.79 

n=l,033 


.80 

n= 1,258 


.74 

n=430 


.71 

n=444 


.73 

n=557 


.76 

n=987 



Test-Re test 

As a measure of reliability over time (stability), participants at three of the schools (a 
comparison elementary school, the comparison middle school, and the pilot high school) 
completed the AEL MASC twice, with a two-week interval between administrations. Table 1 1 
shows the results of Pearson product moment correlations for each factor by participants’ 
original test and subsequent retest scores for the full group and by subgroups. This table shows 
an overall pattern of stability for the entire AEL MASC and its four identified subscales. 
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However, the Family Expectations factor held up least well over time, with correlations ranging 
from .44 to .54. Overall, these correlations indicate a medium relationship (Hinkle, Wiersma, & 
Jurs, 1998). All analyses were significant at the .01 level, which is not unexpected given the 
large sample size. 



Table 1 1 : Test-Retest Pearson Product Moment Correlations 
by Full Group and Subgroups for AEL MASC 





Full 

Group 


Gender 


Ethnicity 


Building Level 


Males 


Females 


White 


Af. Am. 


Elem. 


Mid. 


High 


All items on 
AEL MASC 


.68* 

n=763 


.67* 

n=362 


.73* 

n=383 


.70* 

n=503 


.74* 

n=157 


.69* 

n=44 


.74* 

n=293 


.63* 

n=426 


Factor 1: 

Student 

Belonging 


.76* 

n=762 


.74* 

n=362 


.78* 

n=382 


.76* 

n=503 


.80* 

n=157 


.78* 

n=44 


.77* 

n=293 


.74* 

n=425 


Factor 2: 

Family 

Expectations 


.52* 

n=754 


.48* 

n=358 


.53* 

n=380 


.54* 

n=500 


.46* 

n=157 


.46* 

n=44 


.44* 

n=292 


.54* 

n=418 


Factor 3: 
Student 
Academic 
Efficacy 


.66* 

n=760 


.65* 

n=361 


.68* 

n=382 


.65* 

n=503 


.68* 

n=156 


.68* 

n=44 


.70* 

n=293 


.62* 

n=423 


Factor 4: 
Family/SchooP 
Student 
Involvement 


.71* 

n=760 


.67* 

n=362 


.75* 

n=382 


.71* 

n=503 


.76* 

n=157 


.66* 

n=44 


.78* 

n=293 


.61* 

n=423 



* Significant at .01 



Caution 



Readers should interpret these results with caution. Because these results are based on 
data from one county school district in one state, they are not generalizable to a wider 
population. Although the data reported herein certainly suggest that the AEL MASC is valid and 
reliable, it is not clear whether this might be the case if the instrument were administered in other 
locales or other types of schools. 



- 16 - 





CONCLUSIONS 



The AEL MASC possesses face and concurrent validity, as well as evidence of construct 
validity. It is an instrument that assesses what it purports to measure, and it is correlated with 
another instrument assessing similar constructs. 

The AEL MASC possesses test-retest reliability, in that it measures the relevant 
constructs consistently over time. Internal consistency of the AEL MASC and its four subscales 
has been verified. Thus, the items and subscales measure the constructs reliably. 

The AEL MASC includes four fairly robust subscales or factors: Student Belonging, 
Family Expectations, Student Academic Efficacy, and Family/School/Student Involvement. 
Taken together, these subscales assess the degree to which students think that their schools and 
families provide them with academic nurturance and support, and the extent to which students 
view themselves as intellectually capable. 

The field-test administration of the AEL MASC identified differences in students’ 
perceptions that they belonged at school, their family had high expectations for them, they were 
academically efficacious, and their family was involved in their school life. Females had higher 
perceptions for each of these four areas than males. Further, elementary students had higher 
perceptions for each area than middle and high school students; middle school students also had 
higher perceptions than high school students in each area. 

The AEL MASC has the potential for use in planning school improvement efforts, 
gauging school climate, and investigating factors contributing to achievement gaps based on 
gender or race, especially in the district where this field test was conducted. 

African American students’ perceptions were higher on all four subscales of the AEL 
MASC than those of their White counterparts. In other words, their sense of belonging, family 
expectations, and academic efficacy, as well as the level of family/school/student involvement, 
were all slightly more favorable. Their more positive sense of academic efficacy is somewhat 
surprising, given the achievement gap between White and minority students in this district — their 
perceptions of academic efficacy do not seem to be linked to equivalent or higher academic 
performance. 

Griffard and Wandersee (1999) suggest that teachers may, consciously or otherwise, 
promote “confidence without competence” in their African American students by making them 
feel good about themselves yet failing to nurture them intellectually. Raspberry (2003), in a 
discussion of Ogbu’s 2003 Shaker Heights study, notes the achievement gap may actually be an 
“effort” gap. Ogbu found that even though the African American children in this district 
outperformed other African American youth in the state and much of the nation, they still lagged 
behind their White schoolmates. His research found the African American students put forth less 
academic effort, yet realized the importance of academic achievement. The Shaker Heights 
students did not think they lacked the ability to make better grades; they recognized they could 
do better if they worked harder, but “chose not to” (p. 20). On a similar note, Quaglia and Perry 
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(1993) suggest that student aspirations are composed of both inspiration and ambition. They 
define ambition as the ability to look ahead and invest in the future and inspiration as the ability 
to invest the time, energy, and effort to reach their ambitions. It may be that the academic 
efficacy perceptions of the African American students participating in the field-test 
administration of the AEL MASC were indicating a presence of ambitions yet a lack of 
inspiration. 
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RECOMMENDATIONS 



First, the validity and reliability of the AEL MASC should be verified more robustly with 
data from a larger and more diverse student population. Thus, it is recommended that the 
instrument be field tested again with a student sample from other states, schools, and locale 
types. Such an action would strengthen the instrument and allow for legitimate claims of 
generalizability. 

Second, based on the results of any further field tests, staff should consider eliminating 
the two items that did not load on any of the four subscales or factors. If further analyses 
confirm that neither item contributes meaningfully or statistically to the overall instrument or the 
four subscales, and that the items are unrelated to each other, then this action would be a valid 
refinement of the AEL MASC. 

Third, staff may want to consider other ways of administering surveys to students in the 
future, given the response problems identified through this field test. For example, assigning an 
identification code through the use of bar codes might facilitate any matching of students’ 
surveys for longitudinal studies. 

Fourth, staff may want to explore whether and to what extent the instrument is able to 
discriminate between schools known to encourage academically nurturing climates and those 
known to face challenges in this regard. Such analyses would strengthen claims that the 
instrument accurately and reliably assesses differences between schools. 

Fifth, after the AEL MASC is further validated (perhaps via known groups or 
divergence), future analyses may investigate whether the instrument can statistically identify 
differences among students’ scores in terms of gender, race, or other indicators such as 
achievement level or socioeconomic status. 

Sixth, given the discrepancy between African American students’ higher sense of 
academic efficacy for this administration of the AEL MASC yet their lower overall achievement, 
it may be worth including a sample of student interviews with the next administration of the 
instrument to collect data on relevant issues such as recognition of levels of effort for academic 
achievement and the reasons for putting forth academic effort or not. 
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